[Perf][SM70] Reduce Qwen3.8 no-MTP token latency - #481

Purpose

Reduce Qwen3.8 Flash Next NVFP4 TP4/V100 single-request no-MTP latency without lowering precision. This remains a Draft: the 100 tok/s / 10 ms per token endpoint target is not established by isolated operator timings.

Original task base: 05910abb97446128a259fbd5fbe2bf9ece70a492. Current integration merge base: fbcef6e2f959e95bbe4ca807931abfa2393546e7.

Implemented scope

• Exact real-weight W2 weighted reduction, PLE M=1 plumbing, W13/SwiGLU, shared-expert gate, and QSA output-gate work are recorded in the branch and migration control document.
• Exact TP4 HC compute sharding and isolated HC/MoE communication channels preserve the checkpoint-FP16/FP32 numerical boundaries.
• Latest HC increment assigns 640 hidden coordinates per rank, computes all four branch gates/mixes locally, and gathers 1,280 rather than 5,120 bytes per peer. No weight copy, dtype change, or new public tuning switch. Older extensions retain the existing gate-sharded route; new-op capability and dispatch stay in the DSO that owns the communicator.
• Added benchmarks/kernels/benchmark_sm70_hc_tp4.py: all 96 real HC weight pairs through registered production dispatch, forced old-route control, CUDA Graph timing, and concurrent auxiliary-stream sum2 correctness.
• Added benchmarks/kernels/benchmark_sm70_hc_full_chain.py: complete semantic HC graph including combine/norm and final projections, with intermediate FP16 bit checks and GPU ownership checks around timing groups.
• Source 0303b82d1e now ports the selected vector/parallel-gate 160-CTA fused up/mix/gather to production binding/facade/model dispatch. It appends 21,120 bytes of private packet/counter storage per rank, keeps legacy HC/MoE layouts unchanged, and uses the existing FP16 HC opt-in with owner-DSO capability fallbacks. No weight copy or new user flag. CPU owner/fallback tests: 20 passed; selected SM70 kernel compiles with31 registers/192 bytes shared/zero stack or spills. The registered GPU gate (complete-chain A/B, 512 auxiliary sum2 replays, post-tag-wrap exactness) is queued; no whole-model claim is made from the prototype.

Registered GPU gate now passed (0303b82d1e): full semantic HC 2.109529 -> 1.994807 ms, saving 0.114722 ms / 5.44%. Three fused samples 1.994807/1.993735/1.996370 ms. Four ranks, 16 changing input cases, all intermediate/final outputs, 512 actual auxiliary-stream sum2 graph replays, and post-timing tag-wrap checks have zero FP16 bit mismatches. Sidecar SHA256 5b1ee678bebf6a8fcdb008d5832cfd8ca3d6978558291ec9fe54ec2b9f6cf1bf; result SHA256 b9524acfe04ea92ca3836a404ae590284dc6ea0b8ee4ddfd3a3488e9653a9996. No full model started. The 1.5-ms goal and whole-model output/trace gate remain unachieved; a distinct exact down/gather packet prototype is the next screen.

Latest follow-up (8f74e4b88b): reject the exact down/gather packet fusion (1.989379 -> 2.218926 ms, bitwise but slower). Admit norm-weight early loading only for SM70/FP16/N1/HC4/H2560, leaving prefill and other shapes unchanged. Registered full-HC norm A/B, with up fusion fixed: 1.982525 -> 1.944255 ms (save0.038270ms/1.93%), all four ranks x16inputs plus post-timing outputs bitwise. Five targeted norm dispatch tests pass; the previous 20 owner/fallback tests remain the up-port evidence.

A whole-model attempt selected fused HC during graph capture but the quality harness failed before generation on chat-tokenizer Mapping-vs-list handling; all owned model workers were shut down. This is recorded, not claimed as quality or trace success. CPU preflight now validates explicit integer chat IDs before LLM creation, and the matching installed QdstrmImporter successfully converts old raw traces. The old HC core bucket is reproduced as2.658072ms; complete semantic attribution including final-mixer work is2.701252ms rank-average GPU service. A corrected single-load natural-quality/baseline/trace run is queued. The1.5ms whole-model goal and final256K quality gate remain unachieved.

Full HC <= 1.5 ms follow-up

The user's next target is the complete HC chain below 1.5 ms/token, without precision reduction, accepted with a matched whole-model trace and quality gate. This target is not yet achieved.

At frozen kernel source 50f9fbe374, the portable complete-HC microbenchmark measures old gate-sharded 2.277540 ms versus current hidden-sharded 2.111058 ms (three samples 2.109891/2.111399/2.111058). All intermediate/final outputs pass bitwise over 16 changing inputs on four ranks. This is not a 2.658 -> 2.111 ms whole-model speedup: the old trace used a narrower name-based bucket, and the isolated workload excludes core attention/MoE/PLE computation.

New losing variants are recorded, not admitted: exact combine/norm + down fusion with 0/4/40 weight-chunk prefetch, vector-packed down weights, and joint down/up packing. Up-only packing saves just 0.013476 ms in a paired screen and is not admitted on this evidence alone. A physical down expansion passed bitwise but its contended timings are invalid. No production numerical contract changed, and no full model was started for these screens.

The next experiment targets hidden-sharded up/local mix/output gathering with private per-CTA epochs, distinct from the previously rejected branch-sharded fusion with a global completion counter. The implementation remains local until exactness and complete-chain gains are demonstrated.

That follow-up now has results: 80-CTA local/fused 2.265607/2.180970 ms versus 2.105945 ms control; 160-CTA local/fused 2.161794/2.103446 ms and 320-CTA 2.139696/2.183004 ms versus 2.108150 ms control. The best net gain is only 0.004704 ms; none is admitted. Initial bitwise checks and post-generation-146593 checks pass on four ranks. An LDG128 revision passed its initial exactness check but its timing was rejected when another GPU task entered. The runner now honors that suite's separate reservation across model restarts. A final vector-load/parallel-branch-gate screen is compiled and queued, not a production result.

Latest prototype result: the queued vector-load/parallel-branch-gate screen completed successfully. Complete-HC control 2.108826 ms -> 160-CTA fused 1.999374 ms, saving 0.109452 ms / 5.19%. Three fused samples: 1.999995/1.999374/1.998002 ms. Local-only is 2.068084 ms; 80-CTA fused is 2.082618 ms and is not selected. All four ranks pass intermediate/final bitwise checks over 16 changing inputs and after generation146593 (two packet-tag wraps). There is no packed-weight copy or numerical-contract change. This remains an artifact prototype awaiting production dispatch/capability fallback and auxiliary-stream sum2 validation, then a consolidated whole-model trace/quality gate. It does not establish a 2.658 -> 1.999 ms full-model reduction or satisfy the 1.5-ms goal. All task-owned tests/queues have exited.

HC results and rejected directions

All values below are 96 Mix-only graph calls; they exclude combine/RMSNorm and are not full-HC trace service sums or endpoint TPOT.

The first publication version was also rejected (2.229951 ms). Hidden tile alternatives were bitwise but slower. All screen variants passed 96 real weight pairs × 16 changing inputs × four ranks with zero FP16 bit mismatches. Three paired timing groups use 150 replays each after 1,000 warmups. Raw local evidence is indexed in the migration control document.

Committed production result (aaf63696b6): registered old/new dispatch measures 1.738595 → 1.690003 ms, saving 0.048592 ms / 2.79% per 96 Mix calls. Three samples are old 1.738315/1.739291/1.738595, new 1.689020/1.690590/1.690003; ranges are below 0.1%. This supersedes the prototype's 0.041-ms projection, not the existing endpoint baseline.

Test plan and current results

• Source-matched SM70 custom-AR extension: compiled successfully.
• .venv/bin/python -m pytest -q tests/distributed/test_custom_all_reduce_dispatch.py: 13 passed, including four communicator-DSO ownership/capability combinations.
• Ruff on the six affected Python source/test/benchmark files: passed. Changed-line clang-format and git diff --check: passed.
• Production registered-op gate: passed, all 96 real weight pairs × 16 changing inputs × four ranks, plus 512 graph replays overlapping actual sum2 on an auxiliary stream. HC block/injection and sum2 outputs all have zero FP16 bit mismatches. The owner-aborted initial attempt and lock waits are not claimed as numerical/performance results.
• .venv/bin/python -m pytest -q tests/models/qwen4_exp/test_sm70_fp16_gemv.py -k hc_up_hidden_shards: 1 passed, 18 deselected.
• Runtime: Torch 2.10.0+cu128, CUDA 12.8, Tesla V100-SXM2-32GB TP4; sidecar compiled by NVCC 12.0.140. Binary SHA256 a1fa27c23aea3ee2a7030017ee404c9d2bcb1f3c03889461a070c1f4daded4dd. Raw result production_result.json SHA256 6bf2c047430e586bf1814fbf8ae0fd09a335d4c59decc8d6fff4c5ca6aa37750; logs indexed in the worklog.
• No new full-model startup for this HC screen. The existing matched 8K/513 baseline is 85.14 tok/s / 11.746 ms; the 0.049-ms isolated increment does not establish a new endpoint baseline. All task-owned tests/lock holders exited; unrelated model workers and API were not stopped.
• Before endpoint promotion: one consolidated no-MTP performance/quality gate with fixed non-HC knobs, natural-output quality, and required 256K boundary. Keep earlier overlapping shared-expert savings separate from critical-path TPOT.

Non-duplication and review

This continues existing PR #481, not a second HC/per-token-latency PR. Upstream SGLang/FlashInfer ideas were used as design references; architecture-specific persistent atomic or Blackwell implementations were not copied into the SM70 path. Losing experiments remain out of production dispatch.

AI assistance (OpenAI Codex) was used for implementation, operator experiments, and documentation. Human review is required before merging or endpoint promotion. Commits are DCO-signed.